.TH nx_util 1 LOCAL 
.SH NAME 
nx_util \- A tool to parse & analyze naxsi logs
.SH SYNOPSIS 
.B nx_util [-hoi] [-l
.I FILE
.B ] [-H 
.I DIR
.B ] [-f 
.I filter
.B ] 
.SH DESCRIPTION 
.B nx_util
processes nginx-naxsi log files to generate white-lists or html reports.
It stores NAXSI_FMT/NAXSI_EXLOG events to a sqlite database, and use it to
generate both whitelists and reports.
The user can supply filters to reduce false positives or reduce 
the scope of the html reports.
.SH OPTIONS
nx_util supports options for the three different functions : 
.br

.B Importing filtered events
(from logs, gzipped logs or stdin), 
.br
generating
.B naxsi whitelists
from events, 
.br
generating 
.B "html reports"
representing activity over the specified period.

.br
\&	nx_util -d mysite -l
.I /var/log/nginx/mysite.error.log
.br

This tells nx_util to parse the nginx-naxsi error log 
.I /var/log/nginx/mysite.error.log
, extract NAXSI_FMT and NAXSI_EXLOG events, 
and store them into the sqlite database 'mysite'.
.br

\&	nx_util -d mysite -o
.br

This tells nx_util to display to stdout generated whitelists
from database mysite.

\&	nx_util -d mysite -H 
.I mysite.report
.br

This tells nx_util to generate a html report of events from mysite,
putting the resulting files to the directory
.I mysite.report


\&	nx_util -l /var/log/nginx/*error*log* -f "country = FR"
.br

nx_util will import events from all files matching the
.I /var/log/nginx/*error*log*
string,
.br
but will only import events for which originating country is France.

You can find detailed notes about filters in the
.B FILTERS
section below.



.IP "-l FILES"
Process supplied nginx-naxsi log files.
.br
Space-separated list of files and regular expressions as well.
If no file name is specified, stdin is used to read log lines.
.IP "-H DIR"
.br
Outputs a html static report to the directory.
.br
Python-geoip is required for the world-map section.

.IP "-f FILTER"
.br
Specify one or multiple filters to apply to events.
.br
The 
.B ip,uri,date,server,zone,var_name,id,content,country
keywords can be used,
.br
along with the operators 
.B = != =~ (and >,>,<=, >= for dates)
.br
.br
Note that the
.I python-geoip
is required for country-filters.
.IP "-o"
.br
Outputs generated whitelists to stdout.
.br
If NAXSI_EXLOG datas are present, they will be integrated to output.

.IP "-d db_name"
.br
Specify the name of the sqlite3 database to use.
.br
By default, nx_util will append data to the database called naxsi_sig
in the current directory.
.IP "-c config-file"
.br
config-file specifies directories for sqlite databases and static files used to generate reports. It provides path to naxsi rules as well, to embellish whitelists output.

.SH FILTERS

nx_util's offers a very
.B primtive
language for filtering events that aims at 
.br
.B Lower
false-positive rate when doing learning (by restricting events on country, periods etc.)
.br
Provide
.B focused
reports whenever you whish to investigate a specific event.
.br

Filters need to be supplied with
.B -f
argument, are quoted, and support various keywords : 
.B ip, date, server, uri, zone, var_name, id, content, country
, as well as some simple operators :
.B = != <= >= =~

.B \&		Supported keywords
.br

.B ip
is a string representation of the client ip, and supports =, !=, =~ operators :
.br
\&	-f 'ip = 8.8.8.8'  :
.B Only events from IP 8.8.8.8 will be analyzed.
.br
\&	-f 'ip =~ 8.*'       :
.B Only events from IPs starting by a '8' will be analyzed.
.br
\&	-f 'ip != 1.1.1.1'   :
.B Events from 1.1.1.1 will not be analyzed.
.br

.B date
is a string representation of the date, in the format YYYY-MM-DD HH:MM:SS.
.br
Note that
.B lastweek
and
.B lastmonth
values can be used as shortcuts for now() - (60*60*24*7) and now() - (60*60*24*30).
.br
As well, date supports the > and < operators :
.br
\&	-f 'date > lastmonth and date < lastweek'
will select events that are newer than 30 days and older than 7 days.
.br

\&	When using a full date for comparisons, it needs to be quoted :
.br
\&	-f 'date >= "2013-01-01 00:00:00"'
will select events newer than 1st Jan of 2013.
.br

.B server
corresponds to the "Host" http header.
.br

.B uri
corresponds to the requested uri.
.br
\&	-f 'uri =~ ^/.*foo$' will select events whith an url starting with a '/' and ending with 'foo'.
.br

.B zone
corresponds to the zone in which the event happened. It can be useful when troubleshooting specific events.
.br
\&	-f "zone = BODY" will only select events that happened in a POST/PUT body.
.br

.B id
corresponds to the naxsi-rule ID that matched. It supports >, <, >= and <= operators.
.br

.B var_name
corresponds to the variable name in which the event occured.
.br

.B content
can only be used when importing naxsi_exlog events.
.B Content
refers to offending data captured from the http request.
.br

.B country
is the two-letter country code representation of the client's IP. It requires the GeoIP module to be used :
.br
\&	-f "country = FR" will only select events coming from France.

.SH WHITELISTS
.br
Option
.B -o
is used to generate whitelists from events. Typical output looks like this :
.br
\&	# total_count:1420 (10.66%), peer_count:130 (35.62%) | mysql keyword (|)
.br
\&	BasicRule wl:1005 "mz:$HEADERS_VAR:cookie";
.br

Each suggested whitelist has two line :
.br

The first one indicates the total number of times this event was triggered, its ratio amongst the total number of exceptions,
.br
\&	along with the number of unique peers that triggered the event, and the ratio amongst all unique peers that triggered events.
.br
\&	if your nx_util.conf file contains a valid path to naxsi rules, you will as well get a human readable explenation of the event.
.br

The second line is the whitelist itself, in a format that naxsi can understand.
.br

Those whitelists should be included (either directly or via nginx's include directive) to your configuration,
.br
directly into the location where naxsi is activated.
.br

If you enable $naxsi_extensive_log in your location, nx_util will include extra data in the whitelists, such as :
.br
\&	# total_count:2 (0.27%), peer_count:1 (5.0%) | , in stuff
.br
\&	#exemple (from exlog) : 'Korea, North'
.br
\&	BasicRule wl:1015 "mz:$URL:/user/register|$BODY_VAR:field_country[und]";
.br

After the original comment line, another one was added. The quoted string corresponds to the actual content
.br
of the variable that triggered the exception, helping you to distinguish false positives.

.SH EXAMPLES
.br
cat foobar.log | nx_util -l -o -H test1
.br
\&	nx_util reads events from stdin, then generates whitelists to stdout
.B (-o)
and html report to directory "test1"
.B (-H) 
.br

nx_util -l /var/log/nginx/*error.log -H test1 -f "date > lastweek"
.br
\&	nx_util will read all log files from
.I /var/log/nginx directory
 and create a html report of
.B last week
events to directory "test1"
.br

nx_util -d allsites -i -l /var/log/nginx/*error.log -H test1 -f "date > lastweek"
.br
\&	nx_util will
.B append
last week events from all files in
.I /var/log/nginx/*error.log
to the database
.B allsites.

.SH BUGS

The filters mechanism is extremely primitive and might be subject to bugs if you attempt to create some complex filters.
